The present invention relates to a multiplexer provided with a function for multiplexing a plurality of kinds of media information, a multimedia communication apparatus provided with this multiplexer, and a time stamp generation method used in the multiplexer.
As a system for respectively coding a plurality of items of media information of different types, such as video, audio, data or the like, followed by multiplexing and transmitting such information, an MPEG (Moving Picture Experts Group) system is known.
For example, the in MPEG-2 system, a communication apparatus on the sending side encodes a plurality of items of the media information such as video data, audio data or the like while maintaining respective linkages to generate encoded elementary streams. Next, these encoded elementary streams are converted into packets which are referred to as PES (Packetized Elementary Streams). Each PES is a packet having a variable length and comprises a PES header and a PES payload. The PES header includes a packet start code, a packet length, a flag, control information, PES header length, and conditional coding information. The conditional coding information includes a presentation time stamp (PTS). The PTS is time information required for the device on the receiving side for displaying and outputting a plurality of items of media information such as the video data or audio data while maintaining a time linkage.
Next, each of the above PES is divided into a fixed length as shown in FIG. 6. Then, this divided PES is subsequently inserted into each of the TS packets of a transport stream (TS). The TS packet is a packet having a fixed length, and comprises 4-byte TS header, an adaptation field, or a payload or both. In the adaptation field, a control signal is inserted which is required for the transmission and staffing of information such as PCR (Program Clock Reference) or the like.
Into the TS header, as shown in FIG. 7, a synchronous byte, a transport error indicator, a payload unit start indicator, transport priority information, packet identification information (PID: Packet Identification), transport scramble control information, an adaptation field control flag, and a cyclic counter are inserted in order. Among them, the adaptation field control flag represents the presence or absence of the adaptation field or the payload. The PID is a number for identifying the PES, and is represented by 13 bits. One PES is divided and inserted into the payload of a plurality of TS packet having the same PID to be transmitted. The transport scramble control information represents the presence or absence of the scrambling with respect to the TS payload.
On the other hand, the communication apparatus on the receiving side separates a TS packet into which the video PES and audio PES constituting a program which the user desires to watch and listen to are inserted from the transport stream transmitted from the communication apparatus on the receiving side. Then, the video PES and audio PES of this separated TS packet are respectively decoded with a decoder, and this decoded video data and audio data are reproduced by taking time correspondence therebetween. The time correspondence between this video data and audio data is taken on the basis of the PTS inserted into the PES header.
However, in the video encode scheme, for example, like the encode scheme represented by MPEG-4 Visual, there is an encode scheme in which an irregular frame skip is likely to be generated in a frame of the video data which is encoded. When the encode scheme of this type is used, there is a case in which a correct PTS cannot be added to the PES after conversion in a multiplexer under the influence of the frame skip.
Here, the frame skip refers to a phenomenon in which the encoding with respect to several frames is omitted. This frame skip is generated, for example, in the case where a target bit rate cannot be obtained due to an increase in code quantity generated in encoding. In the absence of the skip frame, there is a case in which a remarkable lowering of the allocated code quantity is caused. In particular, the encode scheme regulated with MPEG-4 Visual is an ultra-low bit rate encode scheme, so that the irregular frame skip is likely to be generated.
The conventional multimedia communication apparatus presupposed on MPEG-2 system does not have a constitution in which the frame skip is considered. As a consequence, with respect to the encoding video data after the generation of the frame skip, a correct time stamp cannot be added. Consequently, in the communication apparatus on the receiving side, video data cannot be reproduced by allowing the video data to take time correspondence to the other media information such as audio data or the like.